Connecting the Dots Using Contextual Information Hidden in Text and Images

The publicly available data feeds are increasing exponentially providing a massive source of intelligence, ironically this plethora of information is what makes succinct details hidden during the analysis of an event of interest. Traditional search engines help narrow down the scope of analysis to a specific event but it is not an easy task for an analyst to study massive amount of relevant documents and get a complete understanding of how each sub-event compose a complex set of interactions between the actors involved in a major event. Here, we describe a framework called Storyboarding that provides summarizations of an event as chains of coherent documents and relevant image objects using publicly available data. Summarization of an event involves the task of identifying relevant entities (e.g., person and location), discovering non-obvious connections between the entities to construct a coherent story – a task which sometimes is referred in the literature as “connecting the dots”, and providing relevant information (e.g., images) to explain the connections better. Our approach provides a mechanism to build a story between two news articles published at two different times using a sequence of intermediate published articles. The storyboarding framework builds a knowledge base first, which helps in providing an image context for each story. A straightforward way to incorporating images during summarization of events is to generate the story using only a similarity network of news articles, as done in Storytelling (Hossain et al. 2012), and then attach images to enrich the presentation of the story. This straightforward approach does not take contents of the images into account whereas the images are likely to carry vital information for a story-building process. In our Storyboarding framework, we leverage image objects (e.g., faces) as added pieces of information to the text content. As an example of our Storyboarding output, Figure 1 and Section 3.3 explain a sub-event of the Boston Marathon Bombing tragedy. The main contributions of the storyboarding framework are (1) a knowledge base that connects textual information with facial features using a probabilistic technique, (2) a Frontalization mechanism that enhances face feature extraction by complementing conventionally used techniques,